29 research outputs found
Evaluating Word Embeddings in Multi-label Classification Using Fine-grained Name Typing
Embedding models typically associate each word with a single real-valued
vector, representing its different properties. Evaluation methods, therefore,
need to analyze the accuracy and completeness of these properties in
embeddings. This requires fine-grained analysis of embedding subspaces.
Multi-label classification is an appropriate way to do so. We propose a new
evaluation method for word embeddings based on multi-label classification given
a word embedding. The task we use is fine-grained name typing: given a large
corpus, find all types that a name can refer to based on the name embedding.
Given the scale of entities in knowledge bases, we can build datasets for this
task that are complementary to the current embedding evaluation datasets in:
they are very large, contain fine-grained classes, and allow the direct
evaluation of embeddings without confounding factors like sentence contextComment: 6 pages, The 3rd Workshop on Representation Learning for NLP
(RepL4NLP @ ACL2018
Multi-Multi-View Learning: Multilingual and Multi-Representation Entity Typing
Knowledge bases (KBs) are paramount in NLP. We employ multiview learning for increasing accuracy and coverage of entity type information in KBs. We rely on two metaviews: language and representation. For
language, we consider high-resource and lowresource languages from Wikipedia. For representation, we consider representations based on the context distribution of the entity (i.e., on its embedding), on the entity’s name (i.e., on its surface form) and on its description in Wikipedia. The two metaviews language and
representation can be freely combined: each pair of language and representation (e.g., German embedding, English description, Spanish name) is a distinct view. Our experiments on entity typing with fine-grained classes demonstrate the effectiveness of multiview learning. We release MVET, a large multiview – and, in particular, multilingual – entity typing dataset we created. Mono- and multilingual finegrained entity typing systems can be evaluated on this dataset
LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning
In recent years, there has been significant progress in developing
pre-trained language models for NLP. However, these models often struggle when
fine-tuned on small datasets. To address this issue, researchers have proposed
various adaptation approaches. Prompt-based tuning is arguably the most common
way, especially for larger models. Previous research shows that adding
contrastive learning to prompt-based fine-tuning is effective as it helps the
model generate embeddings that are more distinguishable between classes, and it
can also be more sample-efficient as the model learns from positive and
negative examples simultaneously. One of the most important components of
contrastive learning is data augmentation, but unlike computer vision,
effective data augmentation for NLP is still challenging. This paper proposes
LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language
Models, which leverages prompt-based few-shot paraphrasing using generative
language models, especially large language models such as GPT-3 and OPT-175B,
for data augmentation. Our experiments on multiple text classification
benchmarks show that this augmentation method outperforms other methods, such
as easy data augmentation, back translation, and multiple templates.Comment: 10 pages, 1 figure, 8 tables, 1 algorithm Proceedings of the 61st
Annual Meeting of the Association for Computational Linguistic
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference
It has been shown that NLI models are usually biased with respect to the
word-overlap between premise and hypothesis; they take this feature as a
primary cue for predicting the entailment label. In this paper, we focus on an
overlooked aspect of the overlap bias in NLI models: the reverse word-overlap
bias. Our experimental results demonstrate that current NLI models are highly
biased towards the non-entailment label on instances with low overlap, and the
existing debiasing methods, which are reportedly successful on existing
challenge datasets, are generally ineffective in addressing this category of
bias. We investigate the reasons for the emergence of the overlap bias and the
role of minority examples in its mitigation. For the former, we find that the
word-overlap bias does not stem from pre-training, and for the latter, we
observe that in contrast to the accepted assumption, eliminating minority
examples does not affect the generalizability of debiasing methods with respect
to the overlap bias.Comment: Accepted at EMNLP 202